Systems and methods for monitoring a self driving vehicle

ABSTRACT

Systems and methods for monitoring a self-driving vehicle are presented. The system comprises a camera, a processor, a communications transceiver, a computer-readable medium, and a display device. The processor can be configured to receive an image of a self-driving vehicle from the camera, and vehicle information from the self-driving vehicle. A graphic comprising the image of the self-driving vehicle and a visual representation of the vehicle information is then displayed on the display device. The vehicle information may comprise any or all of vehicle-status information, vehicle-mission information, vehicle-metric information, and vehicle-environment information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/537,195 entitled “Systems and Methods for Monitoring aSelf-Driving Vehicle” that was filed on 26 Jul. 2017, the contents ofwhich are incorporated herein by reference for all purposes.

FIELD

The described embodiments relate to systems and methods for monitoring avehicle, and in particular, for monitoring a self-driving vehicle usinga monitoring device.

BACKGROUND

The use of self-driving vehicles in industrial facilities brings with itthe generation of significant and meaningful data. However, theself-driving vehicles used in industrial facilities generally lacksophisticated user-interfaces or graphical displays. Therefore, much ofthis data is unavailable, or, at the very least, is cumbersome andinconvenient to access.

Many common situations within industrial facilities represent a need forimmediate and convenient access to some or all of the data generated inassociation with the self-driving vehicles. For example, if a vehicleunexpectedly stops while operating in the facility, there may be severalplausible causes, yet, determining the cause in a given situation may bedifficult without taking cumbersome steps to access relevant data.

There remains a need for human operators to quickly and convenientlyaccess the data that is generated in association with self-drivingvehicles used within an industrial facility.

SUMMARY

In a first aspect, there is a device for monitoring a self-drivingvehicle. The device comprises a processor, a camera, a communicationstransceiver for communicating with the self-driving vehicle,non-transitory computer-readable media, and a display device. Thecomputer-readable medium stores instructions that, when executed,configure the processor to receive an image of the self-driving vehiclefrom the camera, receive vehicle information from the self-drivingvehicle, and display a graphic comprising an image of the self-drivingvehicle and a visual representation the vehicle information on thedisplay device.

According to some embodiments, the vehicle information comprisesvehicle-status information.

According to some embodiments, the vehicle-status information comprisesa vehicle operating state.

According to some embodiments, the vehicle information comprisesvehicle-mission information.

According to some embodiments, the vehicle-mission information comprisesa destination location.

According to some embodiments, the vehicle information comprisesvehicle-environment information.

According to some embodiments, the vehicle-environment informationcomprises sensor scan data.

According to some embodiments, the vehicle-environment informationcomprises camera data.

According to some embodiments, the graphic further comprises at leastone input button for receiving commands for controlling the movement ofthe self-driving vehicle.

In a second aspect, there is a method for monitoring a self-drivingvehicle. A monitoring device in communication with a self-drivingvehicle is used to identify the self-driving vehicle. An image of theself-driving vehicle is captured using the monitoring device. A graphiccomprising the image of the self-driving vehicle and a visualrepresentation of the vehicle information is displayed.

According to some embodiments, the method may comprise the preliminarystep of forming a communications connection between the monitoringdevice and a server associated with the self-driving vehicle, whereinidentifying the self-driving vehicle using the monitoring device isbased on identifying the server using the communications connection.

According to some embodiments, the method may comprise the preliminarystep of capturing a preliminary image of a visual identifier tag on theself-driving vehicle, wherein identifying the self-driving vehicle isbased on the visual identifier tag in the preliminary image.

According to some embodiments, the self-driving vehicle is a firstself-driving vehicle, the preliminary image is a first preliminaryimage, the image is a first image, the vehicle information isfirst-vehicle information. The method may further comprise capturing asecond preliminary image of a second visual identifier tag on a secondself-driving vehicle. The second self-driving vehicle is identifiedbased on the second visual identifier tag in the second preliminaryimage. A second image of the second self-driving vehicle is capturedusing the monitoring device, and second-vehicle information is requestedfrom the second self-driving vehicle using the monitoring device. Thedisplayed graphic comprises the first image of the first self-drivingvehicle and the visual representation of the first-vehicle informationin association with the first image, and the second image of the secondself-driving vehicle and a visual representation of the second-vehicleinformation in association with the second image.

According to some embodiments, the vehicle information comprisesvehicle-state information.

According to some embodiments, the vehicle information comprisesvehicle-mission information.

According to some embodiments, the vehicle information comprisesvehicle-environment information.

According to some embodiments, the graphic further comprises at leastone input button for receiving commands for controlling the movement ofthe self-driving vehicle.

In a third aspect, there is a system for monitoring a plurality ofself-driving vehicles. The system comprises a fleet-management system, adisplay terminal, and at least one camera. The display terminal has adisplay device, a processor, and a non-transitory computer-readablemedium storing instructions that, when executed, configure the processorto receive an image of one of the plurality of self-driving vehiclesfrom the camera, identify the self-driving vehicle, receive vehicleinformation, and display a graphic on the display device comprising theimage of the self-driving vehicle and a visual representation of thevehicle information.

According to some embodiments, the instructions may further configurethe processor to receive vehicle information associated with theself-driving vehicle from the fleet-management system based on theidentity of the at least one self-driving vehicle.

According to some embodiments, the instructions may further configurethe processor to determine a location of the self-driving vehicle basedon the image and receive the identity of the self-driving vehicle fromthe fleet-management system based on the location.

BRIEF DESCRIPTION OF THE DRAWINGS

A preferred embodiment of the present invention will now be described indetail with reference to the drawings, in which:

FIG. 1 is a system diagram of a self-driving vehicle according to someembodiments;

FIG. 2 is a system diagram of a system for monitoring a self-drivingvehicle including the vehicle of FIG. 1, according to some embodiments;

FIG. 3 is a system diagram of a system for monitoring a self-drivingvehicle including the vehicle of FIG. 1, according to some embodiments;

FIG. 4 is a diagram of a system for monitoring a self-driving vehicle,according to some embodiments;

FIG. 5 is a diagram of a system for monitoring a self-driving vehicle inan industrial facility according to some embodiments;

FIG. 6 is a diagram of a monitoring device including a display showing agraphic comprising an image of a self-driving vehicle and a visualrepresentation of vehicle information, according to some embodiments;

FIG. 7 is a diagram of a monitoring device including a display showing agraphic comprising a map of a facility with images of a plurality ofself-driving vehicles, and vehicle information associated with eachself-driving vehicle, according to some embodiments;

FIG. 8 is a diagram of a monitoring device including a display showing agraphic comprising an image of a self-driving vehicle and input buttonsfor controlling the vehicle; and

FIG. 9 is a flow diagram depicting a method of monitoring a self-drivingvehicle according to some embodiments.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Referring to FIG. 1, there is shown a self-driving vehicle 100 accordingto some embodiments. The vehicle comprises a drive system 102, a vehiclecontrol system 104, and one or more sensors 106, 108 a, and 108 b.

The drive system 102 includes a motor and/or brakes connected to drivewheels 110 a and 110 b for driving the vehicle 100. According to someembodiments, the motor may be an electric motor, combustion engine, or acombination/hybrid thereof. Depending on the particular embodiment, thedrive system 102 may also include control interfaces that can be usedfor controlling the drive system 102. For example, the drive system 102may be controlled to drive the drive wheel 110 a at a different speedthan the drive wheel 110 b in order to turn the vehicle 100. Differentembodiments may use different numbers of drive wheels, such as two,three, four, etc.

According to some embodiments, additional wheels 112 may be included (asshown in FIG. 1, the wheels 112 a, 112 b, 112 c, and 112 d may becollectively referred to as the wheels 112). Any or all of theadditional wheels 112 may be wheels that are capable of allowing thevehicle 100 to turn, such as castors, omni-directional wheels, andmecanum wheels.

The vehicle control system 104 comprises a processor 114, a memory 116,a computer-readable non-transitory medium 118, and a communicationstransceiver 120, such as a wireless transceiver for communicating with awireless communications network (e.g. using an IEEE 802.11 protocol orsimilar, also known as “WiFi”).

One or more sensors 106, 108 a, and 108 b may be included in the vehicle100. For example, according to some embodiments, the sensor 106 may be aLiDAR device (or other optical/laser, sonar, or radar range-findingsensor). The sensors 108 a and 108 b may be optical sensors, such asvideo cameras. According to some embodiments, the sensors 108 a and 108b may be optical sensors arranged as a pair in order to providethree-dimensional (e.g. binocular or RGB-D) imaging.

The vehicle control system 104 uses the medium 118 to store computerprograms that are executable by the processor 114 (e.g. using the memory116) so that the vehicle control system 104 can provide automated orautonomous operation to the vehicle 100. Furthermore, the vehiclecontrol system 104 may also store an electronic map that represents theknown environment of the vehicle 100, such as a manufacturing facility,in the media 118.

For example, the vehicle control system 104 may plan a path for thevehicle 100 based on a known destination location and the known locationof the vehicle. Based on the planned path, the vehicle control system104 may control the drive system 102 in order to drive the vehicle 100along the planned path. As the vehicle 100 is driven along the plannedpath, the sensors 106, and/or 108 a and 108 b may update the vehiclecontrol system 104 with new images of the vehicle's environment, therebytracking the vehicle's progress along the planned path and updating thevehicle's location. In other embodiments, the vehicle control system 104may rely in part or in whole on a user-defined path.

Since the vehicle control system 104 receives updated images of thevehicle's environment, and since the vehicle control system 104 is ableto autonomously plan the vehicle's path and control the drive system102, the vehicle control system 104 is able to determine when there isan obstacle in the vehicle's path, plan a new path around the obstacle,and then drive the vehicle 100 around the obstacle according to the newpath.

According to some embodiments, the vehicle control system 104 mayadditionally provide a computer server. For example, the processor 114,the memory 116, the medium 118, and the communications transceiver 120may be configured in order to provide a web server (e.g. using HTTP)which may be made available on a local-area network (LAN), wide-areanetwork (WAN, or over the Internet). The server may be configured toprovide components of the systems and methods described herein, such asfor providing vehicle information to a monitoring device.

According to some embodiments, fiducial markers 122 a and 122 b may beplaced on the vehicle 100. The fiducial markers 122 a and 122 b maygenerally be used to visually orient the vehicle 100, for example, basedon an image of the vehicle 100 that includes the fiducial markers 122 aand 122 b. In some cases, an image of the vehicle 100 that includes thefiducial markers 122 a and 122 b may be analyzed in order to determineposition in terms of vertical, lateral, and longitudinal distances suchas (x, y, z) coordinates, as well as orientation in terms of vertical,lateral, and longitudinal angular displacements such as yaw, pitch, androll. This analysis may include analysis of a single fiducial marker inrespect of itself, or of a combination of two or more fiducial markerscompared with each other.

According to some embodiments, two or more fiducial markers may beplaced on each surface of the vehicle 100. The front, back, and two sidesurfaces of the vehicle 100 are not shown in FIG. 1, and, thus, only thefiducial markers 122 a and 122 b are shown.

Additionally, in some embodiments, a label such as a barcode 124 may beincluded on the vehicle 100 in order to provide additional informationto sensors such as optical scanners or RFID readers. For example, alabel may provide connection information such as a uniform resourcelocator (URL) or IP address associated with the vehicle 100.

Referring to FIG. 2, there is shown a system for monitoring aself-driving vehicle according to some embodiments. The system comprisesa self-driving vehicle 100, a camera 212, a processor 214, a memory 216,a non-transient computer-readable medium 218, a communicationstransceiver 220, and a display device 222. According to someembodiments, the system may also comprise an additional sensor orscanner 224, such as an RFID reader, a second camera, or other opticalsensor. Generally, any or all of the components 212 to 224 may beincluded in a single monitoring device. According to some embodiments,any of the components, such as the camera 212, display device 222, orscanner 224 may be external to the monitoring device. According to someembodiments, the monitoring device may be a mobile phone, tablet, orother mobile computing device. According to some embodiments, thecommunications transceiver 220 may form a wireless communicationsconnection 226 with the vehicle 100.

In use, the camera 212 captures an image of the vehicle 100, which maybe processed by the processor 214. If the vehicle 100 includes fiducialmarkers, and the fiducial markers are captured in the image, then theprocessor 214 may determine the unique identification, spatial location,and orientation of the vehicle 100 relative to the camera 212 and/orrelative to the environment in which the vehicle 100 is operating byusing the fiducial markers. If the vehicle 100 does not include fiducialmarkers, the processor 214 may determine the unique identification,spatial location, and orientation of the vehicle via methods known tothose skilled in the art such as optical character recognition (OCR),visual simultaneous localization and mapping (VSLAM), or iterativeclosest-point matching (ICP).

The processor 214 requests vehicle information from the vehicle 100 viathe communications transceiver 220 and the communications link 226. Forexample, a human user may use the monitoring device to requestparticular vehicle information from the vehicle 100. According to someembodiments, the vehicle information may comprise data defined in termsof a spatial reference.

After receiving the vehicle information, the processor 214 may displaythe image of the vehicle on the display device 222, along with a visualrepresentation of the vehicle information that was received. In the casethe vehicle information comprises data defined in terms of a spatialreference, the visual representation of the vehicle information may beprovided relative to the spatial location and orientation of the vehicle100.

Referring to FIG. 3, there is shown a system for monitoring aself-driving vehicle according to some embodiments. Similar referencenumerals as used in FIG. 2 are used in FIG. 3 in order to identifysimilar or analogous components.

The system comprises a self-driving vehicle 300, a camera 312, acomputer terminal 328 having a processor, memory, non-transientcomputer-readable medium, a communications transceiver, and a displaydevice 322, a server 330, and networking equipment 332. The vehicle 300,the camera 312, the computer terminal 328, and the server 330communicate via the networking equipment 332.

The example system shown in FIG. 3 differs from the example system shownin FIG. 2, in that the camera 312 may be remotely located from thecomputer terminal 328 (i.e. the monitoring device). For example, thecamera 312 may be a camera that is mounted within an industrialfacility, such as a security or other monitoring camera. Furthermore,the server 330 may be used to provide a fleet-management system that maystore and manage some or all of the vehicle information for a fleet ofvehicles. According to some embodiments, the system may comprise a fleetof vehicles including the vehicle 300, a plurality of cameras includingthe camera 312, and a plurality of computer terminals including thecomputer terminal 328.

In use, the camera 312 captures an image of the vehicle 300, which maybe processed by the processor 314. If the vehicle 300 includes fiducialmarkers, and the fiducial markers are captured in the image, then thecomputer terminal 322 may determine the spatial location and orientationof the vehicle 300 relative to the camera 312 and/or relative to theenvironment in which the vehicle 300 is operating. For example, if thelocation of the camera 312 is fixed (such as in the case of a securitycamera), the location of the camera relative to the environment isknown. If the vehicle 100 does not include fiducial markers, theprocessor 314 may determine the unique identification, spatial location,and orientation of the vehicle via methods known to those skilled in theart such as optical character recognition (OCR), visual simultaneouslocalization and mapping (VSLAM), or iterative closest-point matching(ICP).

The computer terminal 328 requests vehicle information from the vehicle300 and/or the fleet-management system on the server 330 via thenetworking equipment 332. For example, a human user may use the computerterminal 328 to request particular vehicle information.

After receiving the vehicle information from the vehicle 300 and/or thefleet-management system on the server 330, the computer terminal 328 maydisplay the image of the vehicle on its display device 322, along with avisual representation of the vehicle information that was received.

Referring to FIG. 4, there is shown a system for monitoring aself-driving vehicle, according to some embodiments. The system is shownas comprising two different monitoring devices 428 and 478 for the sakeof explanation. Similar reference numerals as used in FIG. 3 are used inFIG. 4 in order to identify similar or analogous components, andreference numerals offset by 50 are used with respect to the twomonitoring devices 428 and 478.

As shown, the monitoring device 428 comprises a tablet device having anintegrated camera 412 and an integrated display device 422. The camera412 captures an image of a vehicle 400, which is displayed as a graphicon the display device 422. The graphic displayed on the display device422 also includes visual representations of vehicle information that hasbeen received by the monitoring device 428.

In the example shown, the visual representations of the vehicleinformation include the path that the vehicle is currently following432, the boundary of the vehicle's safety field 434, an environmentalfeature or obstacle 436, an indication of the vehicle's current mission438, the estimated time until the vehicle has completed its currentmission 440, and the current state of charge of the vehicle's battery442.

As shown, the spatial orientation of the vehicle's path 432, the safetyfield 434, and the environmental feature 436 are shown relative to thevehicle's orientation. Other aspects such as the current mission 438,the estimated time until the vehicle has completed its current mission440, and the current state of charge of the vehicle's battery 442 arefixed relative to the display device 422. The determination of therelative orientation of each representation can be made statically atthe time of design, or dynamically based on user preference.

As shown, the monitoring device 478 comprises a display device 472 incommunication with an external camera 462. The camera 462 captures animage of a vehicle 400, which is displayed as a graphic on the displaydevice 472. The graphic displayed on the display device 472 alsoincludes visual representations of vehicle information that has beenreceived by the monitoring device 478.

In the example shown, the visual representations of the vehicleinformation include the path that the vehicle is currently following482, the boundary of the vehicle's safety field 484, an environmentalfeature or obstacle 486, an indication of the vehicle's current mission488, the estimated time until the vehicle has completed its currentmission 490, and the current state of charge of the vehicle's battery492. In the example shown in FIG. 4, the identity 494 of the vehicle isalso indicated.

As shown, the graphic is rendered in a plan view, and the vehicle'sorientation has been translated accordingly in the graphic. The spatialorientation of the vehicle's path 482, the safety field 484, and theenvironmental feature 486 are shown relative to the vehicle'sorientation in plan view.

According to some embodiments, any or all of the vehicle information maybe provided by the vehicle and/or the fleet-information system.According to some embodiments, vehicle information pertaining to theenvironmental features and obstacles may be obtained from the vehicle'smap, and/or from sensor scans performed by the vehicle in approximatelyreal-time.

According to some embodiments, a graphic as shown on the display device422, oriented based on the orientation of the vehicle 400 (e.g.isometric view), and a graphic as shown on the display device 472, inplan view, may both be shown on the same display device. For example,both may be shown simultaneously using a split-screen orpicture-in-picture layout, or the display may be toggled between onegraphic and the other.

Referring to FIG. 5, there is shown a facility 550 in which a system formonitoring self-driving vehicles is deployed. Some components of thesystem are not shown in FIG. 5, for example, a server for operating afleet-management system, and communications networking equipment.

A plurality of self-driving vehicles, including the vehicles 500, 502,and 504 are operating in the facility 550. A plurality of cameras,including cameras 512 a, 512 b, 512 c, and 512 d are operating in thefacility. According to some embodiments, the cameras (generally 512) maybe fixed in a location within the facility 550, for example, by beingmounted on a wall. Generally, the cameras may change their orientation,for example, by turning and zooming. A computer terminal 578 that isremote to the cameras and vehicles may be used as a monitoring devicefor monitoring the vehicles.

Referring to FIG. 6, there is shown a monitoring device 628 in the formof a tablet, displaying a graphic comprising the image of a vehicle 600and multiple visual representations of vehicle information.

As used herein, “vehicle information” pertains to any information thatdescribes the vehicle, a state or condition of the vehicle and itssurroundings, the performance of the vehicle, or the use of the vehicle.Generally, vehicle information can comprise vehicle-state information,vehicle-metric information, vehicle-mission information, andvehicle-environment information. The identity of the vehicle 600 asvisually represented by the text 602 is an example of vehicleinformation.

According to some embodiments, vehicle-state information may includeinformation pertaining to an operating state such as “underway todestination”, “paused for obstruction”, “paused for path planning”,“emergency stopped”, “available”, “conducting mission (unavailable)”, or“charging battery”. For example, a vehicle may be underway to adestination and then stop moving. Subsequent to a request for vehicleinformation, a human operator may then be able to determine that thevehicle has stopped because an unexpected obstruction such as a humanpedestrian or another vehicle has crossed the vehicle's path within itssafety field, thereby causing the vehicle to pause. According to someembodiments, vehicle-state information may also include a timeassociated with the operating state. For example, if the vehicle ispaused for an obstruction, a countdown timer may be displayed to showthe expected delay until the vehicle resumes towards its destination. Inthis way, a human operator using a monitoring device can determine thatthe vehicle is not experiencing an emergency stop or a failure-statethat would may require human intervention. The status as visuallyrepresented by the text 604 is an example of vehicle-state information.

Vehicle-state information may include battery charge information, suchas a discrete charge level, a percentage charge level, a “chargerequired” state, or a “charge recommended” state. Furthermore, thevehicle-state information may indicate whether the vehicle is queued fora subsequent mission, or whether it will be available for a yet-to-beassigned mission after completing its current mission. The batterycharge as visually represented by the text 606 is an example ofvehicle-state information.

Vehicle-state information may include diagnostic information pertainingto the vehicle and/or its components and subsystems. For example,sensors within the vehicle may determine an abnormal operatingcondition, such as temperature, component speed (e.g. wheel speed, motorspeed, etc.), electrical characteristics (e.g. voltage, current,impedance), dynamic characteristics (e.g. unexpected acceleration orjerk as measured by an inertial-measurement unit), and fluid pressures.The warning message as visually represented by the text 608 and thesymbol 610 is an example of vehicle-state information.

Vehicle-state information may also include information pertaining to thecurrent operation of the vehicle. The currently-planned path for thevehicle, as visually represented by dotted line 612, and thepreviously-followed path as visually represented by the dotted line 614are examples of vehicle-state information. Similarly, the current safetyfield associated with the vehicle 600, as visually represented by thearea 616 is an example of vehicle-state information.

According to some embodiments, vehicle-metric information may includeinformation pertaining to the performance of the vehicle, such asvehicle speed, time in service (including separate times for thevehicle, its components and subsystems), time since last maintenance,time since last emergency stop, time since last battery charge,utilization rate (e.g. duty cycle based on amount of time spendfulfilling missions), average speed for missions, average speed, pathefficiency (e.g. total distance travelled over a planned path ascompared to displacement from origin to destination), duration of pausesfor obstacles, duration of pauses for planning new paths, and time spenttravelling from the destination of one mission to the origin of asubsequent mission. The time to destination, as visually represented bythe text 618 is an example of information derived from vehicle-metricinformation.

According to some embodiments, vehicle-mission information may includeinformation pertaining to a mission assigned to the vehicle, such as theorigin of the mission, the destination of the mission, tasks associatedwith the mission, the purpose of the mission, time constraintspertaining to the mission, the priority of the mission, whether themission is to be repeated, and the assignor of the mission. The natureof the mission, as visually represented by the text 620 and thedestination as visually represented by the text 622 are examples ofvehicle-mission information. Similarly, the current distance along thepath the destination, as visually represented by the text 624 is anexample of information derived from vehicle-mission information.

According to some embodiments, vehicle-environment information mayinclude information pertaining to the temperature, humidity, lighting,communication signal strength (e.g. WiFi signal strength), the number ofother vehicles within a given proximity to the vehicle 600 (or vehiclesmeeting particular criteria based on, for example, vehicle-stateinformation), physical features detected by the vehicle's sensors (e.g.LiDAR scanners), audio signal levels, areas of the facility floor wheretraction problems have been detect, and audio or video (or other images)captured by the vehicles sensors (e.g. microphone, cameras). The WiFistrength, as visually represented by the text 626 is an example ofvehicle-environment information.

Referring to FIG. 7, there is shown a monitoring device 778 in the formof a computer terminal, displaying a graphic comprising the images ofthree vehicles, 700, 702, and 704, for example as may have been obtainedfrom more than one camera, along with visual representations of vehicleinformation associated with each vehicle. The graphic shown in FIG. 7 isbased on a plan-view map 706 of the facility, for example, using the mapof the facility stored on a vehicle and/or a fleet-management system incommunication with the monitoring device 778.

Referring to FIG. 8, there is shown a monitoring device 828 in the formof a tablet, displaying a graphic comprising the image of a vehicle 800and input buttons 850. The input buttons may generally be used tocommand the vehicle, for example, as described in U.S. PatentApplication No. 62/486,936 filed on 18 Apr. 2017, which is herebyincorporated by reference.

According to some embodiments, the input buttons 850 may comprise a“joystick” type control for providing commands to move the vehicle. Forexample, as shown in FIG. 8, the input buttons 850 include a forward(“F”) button, a reverse (“R”) button, a “right” button, and a “left”button.

According to some embodiments, since the image of the vehicle 800represents a proper orientation and spatial location of the vehiclerelative to the camera (e.g. based on any or all of fiducial markers,OCR, VSLAM, and ICP), the input buttons 850 can be displayed on adisplay device 822 of the monitoring device 828 with an orientation thatis determined relative to the orientation of the vehicle.

For example, as shown in FIG. 8, the input buttons 850 are shown suchthat the forward and reverse buttons are aligned with the body of thevehicle in the forwards and reverse directions respectively. Similarly,the right and left buttons are oriented in the right and left directionsof the vehicle. As such, as the orientation of the vehicle with respectto the camera (e.g. the monitoring device 828 that includes anintegrated camera), a human user of the monitoring device 828 can bepresented with input buttons 850 that allow intuitive control of thevehicle, without requiring the human user to translate the orientationbetween themselves and the vehicle. In other words, as exemplified bythe arrow-shaped input buttons in FIG. 8, the arrows will always pointin the proper direction relative to the image of the vehicle 800.

According to some embodiments, the display device 822 of the monitoringdevice 828 may be used to obtain other navigational inputs from a humanuser as well. For example, a user may input a point 852 on displaydevice 822 in order establish a waypoint or goal towards which theself-driving vehicle will travel. It is assumed that the point 852 isintended to correspond to a point on the floor on which the vehicle istravelling (i.e. on the same vertical plane as the vehicle). Therefore,it is necessary to translate the position of the point 852 on thedisplay device 822 to a corresponding point on the floor of the facilityin which the vehicle is operating. This translation can be accomplishedbased on the orientation and special location that was performed (e.g.based on any or all of fiducial markers, OCR, VSLAM, and ICP).

Referring to FIG. 9, there is shown a method 900 of monitoring aself-driving vehicle. The method 900 may be implemented using one ormore computer processors, such as those included in any or all of aself-driving vehicle, a monitoring device, a fleet-management system,and a computer terminal. Non-transitory computer-readable mediaassociated with one or more processors may store computer-readableinstructions for configuring the one or more processors to execute thesteps of the method 900.

According to some embodiments, the method 900 may start at step 910 whena communications connection is formed with one or more self-drivingvehicles. According to some embodiments, the connection may be formedusing a monitoring device such as a tablet or another computer.

In some embodiments, the communications connection may be a WiFiconnection that is made using a WiFi transceiver in the monitoringdevice. For example, in the case that the self-driving vehicle isconfigured to provide a server, the WiFi transceiver in the monitoringdevice may directly connect to a WiFi transceiver in the vehicle inorder to communicate with the server.

In some cases, a monitoring device may form a communications connectionwith one or more vehicles using a network, which may include afleet-management server and/or a computer terminal.

According to some embodiments, the communications connection may beestablished by first obtaining communications networking informationfrom fiducial markers placed on the vehicle. For example, a label may bescanned using a camera or other optical sensor in order to obtainconnection information such as a URL or IP address. After the connectioninformation has been obtained by the monitoring device, the monitoringdevice may form the communications connection based on the connectioninformation.

At step 912, a self-driving vehicle is identified. According to someembodiments, the self-driving vehicle may be identified based on one orall of a communications network address or other communicationsparameter, visual recognition of fiducial markers placed on the vehicleor visual recognition of the vehicle itself, and the location of thevehicle.

For example, in the case that step 910 is employed and the monitoringdevice forms a communications link with the vehicle, an IP address, MACaddress, etc. associated with the vehicle may be used to uniquelyidentify the vehicle. According to some embodiments, the steps 910 and912 may be executed as a single step, in that, if a direct connection isformed between the monitoring device and the vehicle, then the vehiclehas necessarily been identified as a result. In other words, accordingto some embodiments, if the monitoring device is only connected to asingle vehicle, then there is no need to specifically identify thatvehicle in order to distinguish it from other vehicles.

At step 914, an image is captured of the self-driving vehicle. The imageis generally captured using a camera, and may be in the form of a singleimage, or a video (e.g. comprising a series of images). According tosome embodiments, the camera may be embedded in the monitoring device.According to some embodiments, the camera may be externally connected tothe monitoring device. According to some embodiments, multiple externalcameras (e.g. security cameras) may be connected via a network to any orall of the monitoring device, a fleet-management system, and a computerterminal.

According to some embodiments, steps 910, 912, and 914 may be executedas a single step, in that, in the process of capturing the image of thevehicle, connection information may be simultaneously obtained from theimage, and the identity may be therefore available either implicitly orexplicitly.

At step 916, vehicle information is requested. According to someembodiments, the vehicle information may be requested automatically, oras the result of an action by human operator of a monitoring device.According to some embodiments, vehicle information may comprisevehicle-state information, vehicle-metric information, vehicle-missioninformation, and/or vehicle-environment information.

At step 918, a graphic may be rendered and displayed on a displaydevice. According to some embodiments, the display device may beintegrated in a monitoring device that includes a camera, or the displaydevice may be remotely located from the camera. According to someenvironments, the graphic comprises an image of the self-drivingvehicle, and associated visual representations of the vehicleinformation. According to some embodiments, the vehicle information maybe visually represented so that it is spatially located and orientedbased on the spatial location and orientation of the vehicle in theimage.

According to some embodiments, the method 900 may be performed formultiple vehicles simultaneously. According to some embodiments, themethod 900 may be performed using multiple cameras. For example, thesteps of method 900 may be executed in parallel for each of a pluralityof vehicles, or the steps of method 900 may be executed sequentially,with an iteration for each of a plurality of vehicles.

The present invention has been described here by way of example only.Various modification and variations may be made to these exemplaryembodiments without departing from the spirit and scope of theinvention, which is limited only by the appended claims.

We claim:
 1. A device for monitoring a plurality of self-drivingvehicles, comprising: a processor; an external camera in communicationwith the processor and located remote from the plurality of self-drivingvehicles; a communications transceiver in communication with theprocessor for communicating with the plurality of self-driving vehicles;a computer-readable medium in communication with the processor; adisplay device in communication with the processor; thecomputer-readable medium storing instructions that, when executed,configure the processor to: receive an image of at least oneunidentified self-driving vehicle of the plurality of self-drivingvehicles, the image captured by the external camera; identify the atleast one unidentified self-driving vehicle in the image captured by theexternal camera; determine a spatial location and orientation of the atleast one identified self-driving vehicle relative to the externalcamera based on the image captured by the external camera; requestvehicle information associated with the at least one identifiedself-driving vehicle, and in response to the request, receive thevehicle information; generate a graphic comprising the image captured bythe external camera and a visual representation of the vehicleinformation, wherein the visual representation of the vehicleinformation is positioned and oriented within the graphic based on thespatial location and orientation of the at least one identifiedself-driving vehicle relative to the external camera; and display thegraphic on the display device.
 2. The device of claim 1, wherein thevehicle information comprises vehicle-status information.
 3. The deviceof claim 2, wherein the vehicle-status information comprises a vehicleoperating state.
 4. The device of claim 1, wherein the vehicleinformation comprises vehicle-mission information.
 5. The device ofclaim 4, wherein the vehicle-mission information comprises a destinationlocation.
 6. The device of claim 1, wherein the vehicle informationcomprises vehicle-environment information.
 7. The device of claim 6,wherein the vehicle-environment information comprises sensor scan data.8. The device of claim 6, wherein the vehicle-environment informationcomprises camera data.
 9. The device of claim 1, wherein the graphicfurther comprises at least one input button for receiving commands forcontrolling the movement of the at least one identified self-drivingvehicle.
 10. A method for monitoring a plurality of self-drivingvehicles, comprising: capturing an image of at least one unidentifiedself-driving vehicle of the plurality of self-driving vehicles using anexternal monitoring device in communication with the plurality ofself-driving vehicles, the external monitoring device being locatedremote from the plurality of self-driving vehicles; identifying the atleast one unidentified self-driving vehicle in the image captured by theexternal monitoring device; determining a spatial location andorientation of the at least one identified self-driving vehicle relativeto the external monitoring device based on the image captured by theexternal monitoring device; requesting vehicle information fromassociated with the at least one identified self-driving vehicle usingthe external monitoring device, and in response to the request,receiving the vehicle information; generating a graphic comprising theimage captured by the external monitoring device and a visualrepresentation of the vehicle information, wherein the visualrepresentation of the vehicle information is positioned and orientedwithin the graphic based on the spatial location and orientation of theat least one identified self-driving vehicle relative to the externalmonitoring device; and displaying the graphic.
 11. The method of claim10, wherein identifying the at least one unidentified self-drivingvehicle in the image captured by the external monitoring device is basedon at least one visual identifier tag on the at least one unidentifiedself-driving vehicle in the image captured by the external monitoringdevice.
 12. The method of claim 11, wherein the at least one identifiedself-driving vehicle is a first identified self-driving vehicle, theimage is a first image, the vehicle information is first-vehicleinformation, further comprising: capturing a second image of a secondunidentified self-driving vehicle using the external monitoring device;identifying the second unidentified self-driving vehicle in the secondimage captured by the external monitoring device; and requestingsecond-vehicle information associated with the second identifiedself-driving vehicle using the external monitoring device, and inresponse to the request, receiving the second-vehicle information;wherein the displayed graphic comprises the first image and the visualrepresentation of the first vehicle information in association with thefirst image, and the second image and a visual representation of thesecond-vehicle information in association with the second image.
 13. Themethod of claim 10, wherein the vehicle information comprisesvehicle-state information.
 14. The method of claim 10, wherein thevehicle information comprises vehicle-mission information.
 15. Themethod of claim 10, wherein the vehicle information comprisesvehicle-environment information.
 16. The method of claim 10, wherein thegraphic further comprises at least one input button for receivingcommands for controlling the movement of the at least one identifiedself-driving vehicle.
 17. A system for monitoring a plurality ofself-driving vehicles, comprising: a fleet-management system forcommunicating with the plurality of self-driving vehicles; a displayterminal in communication with the fleet-management system; at least oneexternal camera in communication with the display terminal, the at leastone external camera being located remote from the plurality ofself-driving vehicles; the display terminal having a display device, aprocessor, and a computer-readable medium storing instructions that,when executed, configure the processor to: receive an image of at leastone unidentified self-driving vehicle of the plurality of self-drivingvehicles, the image captured by the at least one external camera;identify the at least one unidentified self-driving vehicle in the imagecaptured by the at least one external camera; determine a spatiallocation and orientation of the at least one identified self-drivingvehicle relative to the at least one external camera based on the imagecaptured by the at least one external camera; request vehicleinformation associated with the at least one identified self-drivingvehicle, and in response to the request, receive the vehicleinformation; generate a graphic comprising the image captured by the atleast one external camera and a visual representation of the vehicleinformation, wherein the visual representation of the vehicleinformation is positioned and oriented within the graphic based on thespatial location and orientation of the at least one identifiedself-driving vehicle relative to the at least one external camera; anddisplay the graphic on the display device.
 18. The system of claim 17,wherein the processor is configured, when the instructions are executed,to receive vehicle information associated with the at least oneidentified self-driving vehicle from the fleet-management system basedon the identity of the at least one identified self-driving vehicle.